feat: sensitivity_from_table() -- post-hoc Sobol/Morris from a table (#113) - #118
Merged
Conversation
…113) screen() needs a live, re-runnable simulator and a proper Saltelli/ Morris sample design -- neither of which an arbitrary already-collected ResultsTable has, so its indices can't be computed retroactively from whatever points happen to be in the table. sensitivity_from_table() bridges the gap: fit a cheap surrogate over the table (fit_surrogate, #114), then run screen()'s existing Sobol/Morris machinery against the surrogate's predict() instead of a fresh, expensive evaluation. This directly fixes the motivating failure mode: a symmetric U-shaped factor effect (e.g. VBPCApy's ARD prior strength -- too weak under- prunes, too strong over-prunes, optimum in the middle) has near-zero marginal Spearman correlation despite dominating the variance. Sobol S1 through the surrogate correctly ranks it above a co-varying linear factor in the new test (test_sobol_detects_nonmonotonic_effect). Only continuous factors are screened, matching screen()'s own contract; the surrogate is fit on that same continuous subset so non-continuous keys elsewhere in the table's configs are ignored rather than causing an encoding mismatch. TableSensitivity. surrogate_cv_r2 carries the fitted surrogate's cross-validated accuracy (#114) so callers can judge whether to trust a given observable's indices before acting on them. Closes #113
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #117
This PR is stacked on
feat/114-surrogate-accuracy(#117) -- it usesfit_surrogate()'s newcv_r2(from #114) as a trust signal for the sensitivity result. Base is set to that branch rather thanmain; retarget tomainonce #117 merges (or I'll do it).Summary
screen()needs a live, re-runnable simulator and a proper Saltelli/Morris sample design -- neither of which an arbitrary already-collectedResultsTablehas, so its indices can't be computed retroactively from whatever points happen to already be in a table. That rules out the literal ask in #113 ("reuse whatever Sobol machinery already backsscreen()" applied directly to stored data) -- but there's a standard, sound alternative: fit a cheap surrogate over the table, then runscreen()'s existing machinery against the surrogate'spredict()instead of a fresh, expensive simulator call. Since the surrogate is cheap to query, you get proper variance-based indices without any new evaluations.Why this actually fixes the motivating case
#113 was filed because VBPCApy's ARD prior strength (
hp_va) has a mechanistic U-shaped effect (too weak = no pruning, too strong = over-pruning) that a marginal Spearman correlation shows as near-zero, while a co-varying, purely-monotonic factor (xprobe_fraction) that had nothing to do with the believed-dominant effect showed up as the strongest correlate. The new testtest_sobol_detects_nonmonotonic_effectreproduces exactly this shape (y = (a - 0.5)**2 + 0.1*b,aU-shaped and symmetric,blinear) and confirms Sobol S1 through the surrogate correctly ranksaas dominant (importance > 0.5 vsb's remainder) -- the case a marginal correlation would get backwards.Design
sensitivity_from_table(results, factors, *, method="sobol", surrogate_method="rf", ...): fitsfit_surrogate()on the continuous subset offactors, then callsscreen()with the surrogate's.predictasrun_fn.screen()'s own contract exactly); non-continuous keys elsewhere in the table's configs are simply ignored by the encoder rather than causing a mismatch.TableSensitivity(importance, surrogate_cv_r2)--surrogate_cv_r2(fit_regime_surrogate / RegimeSurrogate should report held-out or OOB accuracy #114) is included directly in the result so callers don't have to separately fit and check accuracy before trusting the indices for a given observable.screen()'s own "at least one continuous factor" message if none remain, rather than letting a degenerate empty-feature surrogate fit produce a confusing sklearn error.Test plan
just cigreen locally (ruff format/lint, mypy --strict, coverage 99.51%, 340 passed -- 8 new)test_sobol_detects_nonmonotonic_effect: reproduces the exact U-shaped-effect failure mode from Post-hoc Sobol/Morris total-effect sensitivity from an existing ResultsTable #113 and confirms it's fixedscreen()'s own messagefit_surrogatelow-accuracy warning